Section: New Results

Probabilistic Logic for Activity Recognition

Participants : Carlos F. Crispim-Junior, François Brémond.

keywords: ProbLog2, Horn clauses, Probabilistic models, activity recognition, uncertainty management

In this line of investigation, we have been working on novel models to associate the robust nature of probabilistic logic to noisy observations and the knowledge representation of ontological languages. Our end goal is to design models that are capable of representing the hierarchical structure of complex events (entities, sub-events and constraints) at the same time they can handle the uncertainty of real-life settings, like noisy observations from the vision pipeline. Currently, knowledge representations underperform under noisy scenarios, while prior work in probabilistic logic has provided support either to reason about uncertainty related to entity recognition (probability of recognizing entity A) or to violation knowledge constraints (relevance of violation of constraint $i$ to model $y$).

This work have been carried out in partnership with KU Leuven and the first results of this joint work have been published on the workshop entitled Assisted Computer Vision and Robotics, which was organized during the 2017 edition of the International Conference on Computer Vision. In this paper we propose BEHAVE, a person-centered pipeline for probabilistic event recognition (Fig. 30). The proposed pipeline firstly detects the set of people in a video frame, then it searches for correspondences between people in the current and previous frames (i.e., people tracking). Finally, event recognition is carried for each person using probabilistic logic models (PLMs, ProbLog2 language). PLMs represent interactions among people, home appliances and semantic regions. They also enable one to assess the probability of an event given noisy observations of the real world. BEHAVE was evaluated on the task of online (non-clipped videos) and open-set event recognition (e.g., target events plus none class) on video recordings of seniors carrying out daily tasks. Results have shown that BEHAVE improves event recognition accuracy by handling missed and partially satisfied logic models.

Future work will investigate how to extend PLMs to represent other types of relations, like temporal relations, and how to incorporate low-level information from deep architectures, like Deep Convolution Neural Networks.

Figure 30. BEHAVE:Behavioral analysis of visual events for assisted living scenarios